System and method of persistent address resolution synchronization

ABSTRACT

A method and apparatus of a device that starts an address resolution service on a network element after a boot-up of this network element is described. In an exemplary embodiment, the network element sends an indication of the boot-up. The network element further sends a request for an address resolution table and receives a reply with the requested address resolution table. In addition, the network element starts the address resolution service using the requested address resolution table.

RELATED APPLICATIONS

This application is a divisional of U.S. patent application Ser. No.15/013,966, filed Feb. 2, 2016, which claims the benefit of U.S.provisional application Ser. No. 62/126,387, filed Feb. 27, 2015, theentirety of which are hereby incorporated by reference.

FIELD OF INVENTION

This invention relates generally to data networking, and moreparticularly, to synchronizing address resolution information betweentwo running network elements.

BACKGROUND OF THE INVENTION

Multi-chassis link aggregation (MLAG) is the ability of two or morenetwork elements to act like a single network element when forming linkbundles. This allows a host to uplink to two switches for physicaldiversity, while still having a single bundle interface to manage. Inaddition, two network elements can connect to two other network elementusing MLAG, with all links forwarding.

In an MLAG topology, if there is switch A as primary and switch B assecondary and the user reboots switch A, this reboot triggers a failoverto switch B, with switch B assuming the role of primary. After switch Acomes back up online, switch A starts MLAG negotiation where it takesthe role as secondary. As switch A boots up, the switch A ports (exceptfor a peer-link port with switch B) start in an error disabled stateduring a delay in the newly formed MLAG peer. This allows that therequired states are synced between MLAG peers and further allows agraceful entry for switch-A into the network as an MLAG peer.

After the delay expires, the ports in switch A change its status to UP.If switch A includes routing capability, switch A uses addressresolution protocol (ARP) to resolve Internet Protocol (IP) addresses tomedia access control (MAC) addresses of device coupled to switch A.However, the ARP table will be empty and will require ARP resolutionsfor the unknown hosts. Since ARP cache is not persisted/synced betweenthe peers, there can be traffic drops till the destination is resolved.

SUMMARY OF THE DESCRIPTION

A method and apparatus of a device that starts an address resolutionservice on a network element after a boot-up of this network element isdescribed. In an exemplary embodiment, the network element sends anindication of the boot-up. The network element further sends a requestfor an address resolution table and receives a reply with the requestedaddress resolution table. In addition, the network element starts theaddress resolution service using the requested address resolution table.

In another exemplary embodiment, the network element starts an addressresolution service after a boot-up of a first processing unit in thenetwork element. The network element periodically synchronizes softwareaddress resolution information between the first and a second processingunit of the network element. The network element further receives anindication of the boot-up of the first processing unit. The networkelement determines differences between the hardware address resolutioninformation and the software address resolution information. The networkelement further reconciles the differences so that the hardware and thesoftware address information include the same address resolutionentries. In addition, the network element starts the address resolutionservice on the first processing unit.

Other methods and apparatuses are also described.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example and notlimitation in the figures of the accompanying drawings in which likereferences indicate similar elements.

FIG. 1 is a block diagram of one embodiment of a system with amulti-chassis link aggregation (MLAG) topology for a pair of networkelements.

FIG. 2 is a block diagram of one embodiment of a pair of MLAG peers thatsynchronize an address resolution table.

FIG. 3A is a flow diagram of a process to synchronize an addressresolution table.

FIG. 3B is a flow diagram of a process to synchronize an addressresolution table.

FIG. 4 is a block diagram of a system with network elements thatparticipate in a Virtual Router Redundancy Protocol (VRRP) topology.

FIG. 5 is a network element with multiple central processing units(CPUs).

FIG. 6 is a flow diagram of one embodiment of a process to synchronizean address resolution table between multiple CPUs.

FIG. 7 is a flow diagram of one embodiment of a process to reconciledifferences between hardware and software address resolution tables.

FIG. 8 is a block diagram of one embodiment of a synchronize module thatsynchronizes an address resolution table.

FIG. 9 is a block diagram of one embodiment of an ARP synchronize andrecovery module that synchronizes an address resolution table betweenmultiple CPUs.

FIG. 10 is a block diagram of one embodiment of ARP resolution modulethat reconciles differences between hardware and software addressresolution tables.

FIG. 11 illustrates one example of a typical computer system, which maybe used in conjunction with the embodiments described herein.

FIG. 12 is a block diagram of one embodiment of an exemplary networkelement that synchronizes an address resolution table of a networkelement.

DETAILED DESCRIPTION

A method and apparatus of a device that starts an address resolutionservice on a network element after a boot-up of this network element isdescribed. In the following description, numerous specific details areset forth to provide thorough explanation of embodiments of the presentinvention. It will be apparent, however, to one skilled in the art, thatembodiments of the present invention may be practiced without thesespecific details. In other instances, well-known components, structures,and techniques have not been shown in detail in order not to obscure theunderstanding of this description.

Reference in the specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin connection with the embodiment can be included in at least oneembodiment of the invention. The appearances of the phrase “in oneembodiment” in various places in the specification do not necessarilyall refer to the same embodiment.

In the following description and claims, the terms “coupled” and“connected,” along with their derivatives, may be used. It should beunderstood that these terms are not intended as synonyms for each other.“Coupled” is used to indicate that two or more elements, which may ormay not be in direct physical or electrical contact with each other,co-operate or interact with each other. “Connected” is used to indicatethe establishment of communication between two or more elements that arecoupled with each other.

The processes depicted in the figures that follow are performed byprocessing logic that comprises hardware (e.g., circuitry, dedicatedlogic, etc.), software (such as is run on a general-purpose computersystem or a dedicated machine), or a combination of both. Although theprocesses are described below in terms of some sequential operations, itshould be appreciated that some of the operations described may beperformed in different order. Moreover, some operations may be performedin parallel rather than sequentially.

The terms “server,” “client,” and “device” are intended to refergenerally to data processing systems rather than specifically to aparticular form factor for the server, client, and/or device.

A method and apparatus of a network element that starts an addressresolution service on a network element after a boot-up of this networkelement is described. In one embodiment, the network elementparticipates as a peer in an MLAG topology, which allows this networkelement and the other peer in the MLAG topology to act as a singlenetwork element. In this embodiment, an MLAG topology allows a host tobe coupled to the multiple network elements for physical diversity,while still only having a single bundle interface to manage. In an MLAGtopology, one of the network elements is a primary and another is asecondary. In one embodiment, the primary perform similar functions(receive and forward data, fulfill address resolution requests, or othernetwork functions), except the primary network element can perform aSpanning Tree Protocol (STP) service, with the secondary network elementacting in a standby mode for the STP service. In one embodiment, thisnetwork element is a primary network element in the MLAG topology andthe other network element is a secondary network element. In addition,these two network elements are first hop gateways as these networkelements couple a layer 2 network with a layer 3 network. In oneembodiment, these two network elements are the first hop gateway fordevices coupled to the layer 2 network. By being the first hop gateway,each of these network elements runs an address resolution service thatreturns a mapping between a layer 2 and a layer 3 address for a coupleddevice. In one embodiment, the address service is an address resolutionprotocol (ARP) service.

In one embodiment, if the primary network element reboots, the secondarynetwork element takes over for the primary network elements. As theformerly primary network element comes back up, this network elementsdoes not have an address resolution table, because this table is notpersistent across a network element reboot. In addition, the ports arein an error disable state as the former primary network element isbooting up. Instead of bringing the ports back to an up state after thereload delay, the formerly primary network element sends a request forthe address resolution table to the formerly secondary (now primary)network element. In addition, the former primary network elementmaintains the ports that are part of the MLAG in the error disablestate. The now primary network element sends the address resolutiontable to the now secondary network element. The now secondary networkelement installs the address resolution table and brings the MLAG portsinto a normal state from the error disable state. This allows the nowsecondary network element to handle address resolution requests with acurrent (or nearly current) address resolution table. The boot upprocess can also be applied to network elements is a Virtual RouterRedundancy (VRRP) topology.

In another embodiment, a network element includes a primary centralprocessing unit (CPU) and a standby CPU. In this embodiment, the standbyCPU takes over the processing unit operations for the primary CPU incase the primary CPU fails or reboots. Prior to the CPU failing, theprimary CPU periodically sends the address resolution table to thestandby CPU. The standby CPU receives the address resolution table andprograms this table. In one embodiment, the primary CPU sends theaddress resolution table every five minutes (or another time period,such as one minute, 30 seconds, 10 minutes, or time period that isgreater, smaller, or different (e.g., the synchronization period can bebased on an ARP aging interval that is configured for the ARP service)).If the primary CPU fails, the standby CPU takes over, including theresponsibility of running the address resolution service. Beforestarting the address resolution service, the standby CPU reconciles theaddress resolution table of the standby CPU with the hardware addressresolution table used by the forwarding engine(s) of the networkelement. The standby CPU starts the address resolution service with thereconciled address resolution table and takes over the other processingunit operations from the primary CPU.

FIG. 1 is a block diagram of one embodiment of a system 100 with amulti-chassis link aggregation (MLAG) topology for a pair of networkelements. In one embodiment, network elements 102A-B couple a layer 3network 106 with a layer 2 network 116. In one embodiment, the layer 3network 106 is a network that forwards data packets using layer 3information contained in those data packets. For example and in oneembodiment, a layer 3 network can be an Internet Protocol (IP) network,in which forwarding decisions are made on the IP address included in thedata packets. In another embodiment, the layer 2 network 116 is anetwork that forwards data packets using layer 2 information containedin those data packets. For example and in one embodiment, a layer 2network can be an Ethernet network, in which forwarding decisions aremade on the media access control (MAC) address included in the datapackets. In one embodiment, these data packets include both IP and MACaddresses, and which address is used for a forwarding decision is basedon how each of the networks is configured. In one embodiment, networkelements 102A-B are coupled with each other over a peer link 108.

In one embodiment, because network elements 102A-B couple the layer 2network 116 with the layer 3 network 106, the network elements 102A-Bare each a first hop gateway to the layer 3 network 106 for devicescoupled to the layer 2 network 116. Thus, each of the network elements102A-B gives access to the layer 3 network 106 for network element 104and devices 112A-B. In this embodiment, network element 104 is a layer 2switching device that couple devices 112A-C to network elements 102A-B.The network elements 102A-B and/or 104 can each be a device thatprovides network access to a network (e.g., physical network,virtualized network, etc.). A network element can be a switch, router,hub, bridge, gateway, etc., or any type of device that can provideaccess to a network. In a further embodiment, the network element 102A-Band/or 104 can be a device that communicates with a network (e.g.,server, personal computer, or another device that can communicate with anetwork). In one embodiment, one or more of the network elements 102A-Band/or 104 can be a physical or virtual device. In one embodiment, thedevices 112A-C can be a server, computer, switch, network element, oranother type of device that can communicate data with a network.

In order to increase the bandwidth availability and redundancy foraccess to layer 3 network 106, network elements 102A-B and 104 can forma multi-chassis link aggregation group (MLAG) between them. In oneembodiment, an MLAG is a way to inverse multiplexing multiple linksbetween network elements to increase bandwidth availability andredundancy. In addition, network elements participating in an MLAG willhave two or more network elements configured to act like a singlenetwork element when forming the link aggregation groups. In thisembodiment, this allows a device to uplink to the two network elements102A-B. In one embodiment, the MLAG 110 includes network elements 102A-Beach having a separate link to the network element 104, respectively.Thus, in MLAG 110, there are two links between the network elements102A-B and 104. However, for each of devices 112A-C, MLAG 110 appears asa single link to the layer 3 network 106.

Because the network elements 102A-B are the first hop gateway fordevices 112A-C and couple the layer 2 network 116 to the layer 3 network106, the network elements 102A-B maintain an address resolution tablethat is a mapping between the layer 2 and layer 3 addresses for each ofthe network element 104 and devices 112A-E as well as other devices inthe layer 2 network 116. The devices 112A-C and network element 104 willeach have two different addresses assigned to them: a layer 3 address(e.g., an Internet Protocol (IP) address) and a layer 2 address (e.g., aMedia Access Control (MAC) address). For a device that wants to send apacket to another device and does not know the layer 3→4 layer 2 addressmapping, the sending device will broadcast a request to determine thecorresponding layer 3 address for the layer 2 address. For example, thesending device broadcasts an ARP address request to local networkelements (e.g., network elements 102A-B) to determine a MAC address thatcorresponds to an IP address. The network element 102A-B can answer theARP request, using an ARP response packet as is known in the art.

The network element 102A-B maintains an association between the layer 3address and a corresponding layer 2 address in a table. For example andin one embodiment, a network element 102A-B uses an ARP table to includeone or more ARP entries. Each entry maintains an association between asource IP and source MAC address. Each association is valid for acertain period of time, after which the association expires. Expirationtimes can range from a matter of seconds to twenty minutes or more.

In the MLAG 110, one of the network elements 102A-B will be a primarymember of the MLAG 110 and the other network element will be a secondarymember. In one embodiment, each of the network elements 102A-B maintainsan ARP table. In this embodiment, the ARP table is not persistent and,thus, if one of the network elements 102A-B reboots, that networkelement will lose this ARP table and resume operations with a blank ARPtable. This can be a big problem if that network element is handlingaddress resolution requests for hundreds or thousands of devices. In oneembodiment, if the primary network element 102A reboots, instead ofbringing up the ports that are art of the MLAG to a normal state, thenetwork element 102A sends a request to the secondary network element102B to synchronize the ARP table and maintains the MLAG ports in anerror disable state. The network element 102B sends a copy of the ARPtable to network element 102A. The network element 102A takes this ARPtable and installs it. With the table installed, the network element102A brings the MLAG ports into a normal state, which allows the addressresolution service using the installed ARP table to resume and, further,resumes MLAG service operation.

FIG. 2 is a block diagram of one embodiment of a system 200 with pair ofMLAG peers 202A-B that synchronizes an address resolution table. In FIG.2, network elements 202A-B each include a synchronized module 206A-B andan ARP table 208A-B. In one embodiment, the synchronize module 206A-Bsends ARP table synchronize requests and fulfills those requests. Inthis embodiment, a network element 202A will send out an ARP tablesynchronize request after network element 202A reboots and wants tobuild an ARP table when configured as an MLAG peer in an MLAG topology.In addition, the ARP table 208A-B is a cache that includes layer 3 tolayer 2 address mapping as well as other information (e.g., expirationtime, or other information for this mapping). In one embodiment, networkelement 202A is a primary in the MLAG topology and network element 202Bis the secondary. In this embodiment, if network element 202A reboots orotherwise goes off-line, network element 202B would become the primarydevice in this MLAG topology.

If the network element 202A reboots, network element 202A would lose theARP table 208A because this table is not persistently stored, after, apersistently stored ARP table is out of date after reboot (especially ifthe network element 202A is down for several hours, days, or longer).Instead of restarting the address resolution service with a blank ARPtable 208A, network element 202A sends an ARP table synchronize requestto network element 202B so as to get a copy of ARP table 208B. Networkelement 202A would install this copy of the ARP table 208B on networkelement 202A, and would use this new ARP table 208A to restart theaddress resolution service. In one embodiment, network element 202Asends the ARP table request to network element 202B. Network element202B receives this request, serializes the ARP table 208B, and sends theserialized ARP table back to network element 202A. Network element 202Areceives the serialized table and installs this table on network element202A as the ARP table 208A to be used to support the address resolutionservice on network element 202A. At this point, network element 202A hasthe same ARP table as network element 202B or a table that is nearly thesame. In one embodiment, the two ARP tables 208A-B may differ as networkelement 202B may receive additional updates to ARP table 208B during thesynchronization process. In this embodiment, for the relatively fewentries that are out of sync, network element 202A would send out ARPrequests as needed to update the ARP table 208A. With the newlyinstalled ARP table 208A, network element 202A starts the addressresolution service.

By installing a copy of the ARP table 208B, network element 202A becomesmore efficient as to starting an address resolution service with a blankARP table because network element 202A would need to send out needlessARP requests. In one embodiment, these extra ARP requests can become abig burden on the network if the ARP table 208B has hundreds orthousands of entries. Without a current, or nearly current, ARP table208A, network element 202A may cause a drop in traffic across the layer2 network because network element 202A will need to build up the ARPentries in ARP table 208A. Thus, by using this synchronization processafter a reboot, this process advances the technical field of networktechnology by eliminating or reducing traffic loss after an MLAG peerreboots. In one embodiment, this process is executed by first hopgateway, in which the first hop gateway is a network element thatstraddles two different networks (e.g., network elements 102A-B thatstraddle the layer 2 network 116 and the layer 3 network 106 asillustrated in FIG. 1 above). Synchronization between these two networkelement peers is further described in FIGS. 3A-B below.

FIGS. 3A and 3B are flow diagrams of processes to synchronize an addressresolution table. FIG. 3A is a flow diagram of a process 300 to requestan address resolution table from a network element peer. In oneembodiment, process 300 is performed by a synchronization module torequest an address resolution table from a network element peer, such assynchronization modules 206A-B as described in FIG. 2 above for asecondary network element in an MLAG topology. In this embodiment,process 300 is performed by a secondary network element in an MLAGtopology. In FIG. 3A, process 300 begins by receiving an indication ofthe boot up by a network element at block 302. In one embodiment, theindication of the boot up for a network element can be an indication tostart an address resolution service, where the network element isconfigured as an MLAG peer with another network element. For example andin one embodiment, network element 202A is peered with network element202B and configured in an MLAG environment with network element 202B.During the reboot, the ports of the network element 202A, except for thepeer link port, are put in the error disable state and remain in theerror disable state until the address resolution table is installed. Inone embodiment, a port in an error disable state does not transmit orreceive data with another device coupled to this port. For example andin one embodiment, a port in an error disable state would not receiveARP requests or serve these requests. At block 304, process 300 sends arequest for an address resolution table.

Process 300 receives the address resolution table from the peerednetwork element at block 306. In one embodiment, the address resolutiontable is a serialized table that is serialized for transmission. Atblock 308, process 300 loads the address resolution table. In oneembodiment process 300 takes the serialized address resolution table,builds the address resolution table entries, and stores these entries inthe address resolution table. For example and in one embodiment, foreach set of layer 3 to layer 2 address mappings (as well as otherinformation), process 300 builds a new address resolution table entry.After processing these entries, process 300 stores the addressresolution entries in the new address resolution table. At block 310,process 300 brings up the peer link ports in an error disable state to anormal state, which allows the MLAG service to start and the addressresolution service to receive and fulfill address resolution requests.In one embodiment, a port in a normal state can transmit and receivedata with another device coupled to the port. In this embodiment, thenetwork element can receive and fulfill address resolution requests viathe port in the normal state.

FIG. 3B is a flow diagram of a process 350 to fulfill an addressresolution table request from a network element peer. In one embodiment,process 350 is performed by a synchronization module to fulfill theaddress resolution table request from a network element peer, such assynchronization modules 206A-B as described in FIG. 2 above. In thisembodiment, process 350 is performed by a primary network element in anMLAG topology. In FIG. 3B, process 350 begins by receiving an addressresolution table request at block 352. At block 354, process 350serializes the address resolution table, so as to make the addressresolution table ready for transmission to the requesting networkelement. In one embodiment, process 350 serializes the addressresolution table by converting the stored address resolution table intoa format that is easier to transmit. Process 350 since the serializedaddress resolution table to the requesting network element at block 356.

While in one embodiment, the address resolution table synchronizationprocess is performed by network elements in an MLAG topology, inalternate embodiments, this address resolution table synchronizationprocess can be performed by network elements in other types of topology.For example and in one embodiment, network elements that are first hopgateways in a virtual router redundancy protocol (VRRP) topology can usethis address resolution table synchronization process to restart andaddress resolution service if one of the network elements reboots (foreither IPv4 or IPv6). FIG. 4 is a block diagram of a system 400 withnetwork elements that participate in a VRRP topology. In FIG. 4, networkelements 402A-B are in a VRRP topology, in which these network elements402A-B are first hop gateways for devices 406A-B and couples layer 2network 404 with layer 3 network 406. In one embodiment, layer 3 network406 is an IPv4 network. The network elements 402A-B can each be a devicethat provides network access to a network (e.g., physical network,virtualized network, etc.). A network element can be a switch, router,hub, bridge, gateway, etc., or any type of device that can provideaccess to a network. In a further embodiment, the network elements402A-B can be a device that communicates with a network. In oneembodiment, one or more of the network elements 402A-B can be a physicalor virtual device. In one embodiment, the devices 406A-N can a server,computer, switch, network element, or another type of device that cancommunicate data with a network.

In one embodiment, because network elements 402A-B are first hopgateways for the layer 3 network 408, network elements 402A-B eachmaintain an address resolution table. Network element 402A is a primaryand network element 402B is the secondary in the VRRP topology. In thisembodiment, if network element 402A reboots, network element 402Bbecomes the primary and network element 402A sends an address resolutiontable request to network element 402B to get a copy of the addressresolution table stored on network element 402B. Network element 402Breceives and fulfills this request by sending a serialized copy of theaddress resolution table to network element 402A. Network element 402Ain a VRRP topology, does not put ports into an error disabled state.Instead, the network element 402A that is a VRRP master reboots andcomes up as a VRRP backup router. The network element 402A waits for theVRRP hello packet from the new VRRP master. If this indicates that thenew VRRP master has a lower priority (which will be the case as thepriority is controlled by configuration), the network element 402Aassumes the role of VRRP master and the network element 402B becomes theVRRP backup. After a reboot, network element 402A performs the addressresolution request and fulfillment process as described above in FIGS.3A and 3B and ignores the VRRP hello packets until the addressresolution synchronization is complete. Once network element 402A hascompleted this synchronization, network element will resume processingthe VRRP hellos from the peer and transition to the master VRRP router.This will reduce or remove any traffic loss. In one embodiment, networkelements 402A-B performs the address resolution request and fulfillmentprocess as described above in FIGS. 3A and 3B.

FIG. 5 is a network element 500 with multiple central processing units(CPUs) 502A-B. In FIG. 5, the network element 500 includes CPUs 502A-Band forwarding engines 504. In one embodiment, the CPUs 502A-B are eacha processing unit that is used to control the processing functions ofthe network element 500. For example and in one embodiment, the primaryCPU 502A controls the configuration of the forwarding engine(s) 504,retrieves the stored monitoring data, retrieves stored statistics,and/or performs other control functions. In one embodiment, the networkelement has a primary CPU 502A and a standby CPU 502B. In thisembodiment, if the primary CPU 502A goes down, the standby CPU 502Btakes and resumes the operations of the previously primary CPU 502A. Inaddition, CPU 502A becomes the standby CPU. Furthermore, the forwardingengines 504 can each receive, process, and/or forward network traffic.In one embodiment, each hardware forwarding engine 504 includes one ormore hardware tables that are used to store configuration data,monitoring data, reporting data, and/or other management data for thehardware forwarding engine 504.

In one embodiment, each of the CPUs 502A-B includes an ARP sync andrecovery module 506A-B, respectively. In this embodiment, the networkelements 504A-B used the ARP sync and recovery module 506A-B tosynchronize the ARP tables between the two CPUs 502A-B. Each CPU 502A-Bmaintains a copy of the ARP table. Periodically, the two CPUs 502A-Bsynchronize the ARP table between the two CPUs 502A-B. For example andin one embodiment, the active CPU 502A maintains and updates the ARPtable based on the information received by the network element 500(e.g., receiving ARP updates from the network, etc.). In this example,the active CPU 502B will periodically send the current ARP table to thestandby CPU 502A, where the standby CPU 502B receives this ARP table andincorporates the table into the state maintained by the standby CPU502B. Alternatively, the standby CPU 502B can periodically request andreceive the current ARP table from the active CPU 502A. In oneembodiment, the periodicity of synchronization can be on the order ofminutes, such as 5 minutes or another period (e.g., the synchronizationperiod can be based on an ARP aging interval that is configured for theARP service). In this embodiment, the standby CPU 502B sends a requestto the active CPU 502A. The active CPU 502A sends a copy of the ARPtable to the standby CPU 502B. The standby CPU 502B stores the ARPtable. By periodically updating the ARP tables instead of continuallyupdating them, the network element operates more efficiently. Intopologies with thousands of devices, there may be a continual churn inthe ARP table entries. By periodically synchronizing the ARP tables, thenetwork element reduces the number of synchronizations that are employedby the network element. In addition, each forwarding engine 504 includesa hardware-stored ARP table (not illustrated).

If the active CPU 502A goes down or restarts, the standby CPU 502B takesover and becomes the active CPU. The now-active CPU, CPU 502B, starts anaddress resolution service using the stored ARP table. Prior to startingthis service, CPU 502B determines if the ARP table stored for the CPU isthe same as the ARP table stored with the forwarding engine(s) 504. Ifnot, CPU 502B reconciles the differences between the two APR tables sothat the two ARP tables are the same. Keeping a software based ARP tablesynchronized is further described in FIGS. 6 and 7 below.

FIG. 6 is a flow diagram of one embodiment of a process 600 tosynchronize an address resolution table between multiple CPUs. In oneembodiment, process 600 is performed by an ARP sync and recovery module,such as the ARP sync and recovery module 506A-B as described in FIG. 5above. In FIG. 6, process 600 beings by periodically synchronizing theARP tables at block 602. In one embodiment, process 600 periodicallysynchronizes the tables. In this embodiment, process 600 receives thecurrent ARP table form the active CPU. Process 600 installs the ARPtable for the standby CPU to use. At block 604, process 600 receives anindication of a CPU failure. In one embodiment, the active CPU fails andprocess 600 receives an indication that this CPU is no longer available.In this embodiment, the standby CPU will take over the CPU operationsfrom the active CPU. Process 600 reconciles the hardware and softwaredifferences in the respective ARP tables at block 606. In oneembodiment, the ARP table currently available to the standby CPU may beout of date as this ARP table is periodically synchronized with the ARPtable maintained by the previously active CPU. There is a currentversion of the ARP table that is maintained by the forwarding engine(s).In this embodiment, process 600 determines if there are any entries thatare different between the hardware ARP table maintained by theforwarding engine(s) and the software ARP table available to the standbyCPU. If there are differences, process 600 updates the software ARPtable so that the software ARP table has the same entries as thehardware ARP table. Reconciling the hardware and software ARP tables isfurther described in FIG. 7 below. At block 608, process 600 resumes therole of the now active CPU. In one embodiment, this resumption ofoperations includes starting and address resolution service using theupdated ARP table.

FIG. 7 is a flow diagram of one embodiment of a process 700 to reconciledifferences between hardware and software address resolution tables. Inone embodiment, process 700 is performed by an ARP sync and recoverymodule, such as the ARP sync and recovery module 506A-B described inFIG. 5 above. In FIG. 7, process 700 begins by retrieving the hardwareand software ARP entries in one embodiment, the hardware and softwareARP entries may be different, because the software ARP table may nothave been recently synchronized. At block 704, process 700 determines ifthe hardware and software ARP entries are the same. If these entries arethe same, process 700 returns an okay at block 706. If there aredifferences, at block 708, process 700 fixes the differences between thehardware and software ARP entries. For example and in one embodiment, ifthere is an entry present in the hardware ARP table that is missing inthe software ARP table, process 700 sends out an ARP request todetermine the missing ARP entry. If there is an ARP entry that ispresent in the software ARP table, but not present in the hardware ARPtable, process 700 deletes the entry from the software ARP table.

FIG. 8 is a block diagram of one embodiment of a synchronize module206AB that synchronizes an address resolution table. In one embodiment,the synchronize module 206AB includes synchronize request module 800 andsynchronize fulfill module 850. In one embodiment, the synchronizerequest module 800 includes receive boot-up indication module 802, sendtable request module 804, receive table module 806, load table module808, and ML start module 810. In one embodiment, the receive boot-upindication module 802 receives an indication of a boot-up as describedin FIG. 3A, block 302 above. The send table request module 804 send anaddress resolution table request as described in FIG. 3A, block 304above. The receive table module 806 receives the address resolutiontable as described in FIG. 3A, block 306 above. The load table module808 loads the address resolution table as described in FIG. 3A, block308 above. The MLAG start module 810 allows the multi-link service tobecome active by putting the multi-link ports into a normal state asdescribed in FIG. 3A, block 310 above.

The synchronize fulfill module 850 includes the receive request module852, serialize table module 854, and send table module 856. In oneembodiment, the receive request module 852 receives a request for anaddress resolution table as described in FIG. 3B, block 352 above. Theserialize table module 854 serializes the address resolution table asdescribed in FIG. 3B, block 354 above. The send table module 856 sendsthe address resolution table as described in FIG. 3B, block 356 above.

FIG. 9 is a block diagram of one embodiment of an ARP synchronize andrecovery module 506AB that synchronizes an address resolution tablebetween multiple CPUs. In one embodiment, the ARP synchronize andrecovery module 506AB includes synchronize module 902, receive failureindication module 904, reconcile table module 906, and resume operationsmodule 908. In one embodiment, the synchronize module 902 synchronizesthe address resolution table with the primary CPU as described in FIG.6, block 602 above. The receive failure indication module 904 receives afailure indication of the primary CPU as described in FIG. 6, block 604above. The reconcile table module 906 reconciles the hardware andsoftware address resolution tables as described in FIG. 6, block 606above. The resume operations module 908 resumes the operations of theCPU as described in FIG. 6, block 608 above.

FIG. 10 is a block diagram of one embodiment of reconcile table module1006 that reconciles differences between hardware and software addressresolution tables. In one embodiment, the reconcile table module 1006includes retrieve entries module 1002, compare entries module 1004, andfix differences module 1006. The retrieve entries module 1002 retrievesthe address resolution entries from the hardware and software addressresolution table entries as described in FIG. 7, block 702 above. Thecompare entries module 1004 compares the hardware and software addressresolution table entries as described in FIG. 7, block 704 above. Thefix differences module 1006 fixes the differences between the hardwareand software address resolution table entries as described in FIG. 7,block 708 above.

FIG. 11 shows one example of a data processing system 1100, which may beused with one embodiment of the present invention. For example, thesystem 1100 may be implemented including a network elements 102A-Band/or 104A-B as shown in FIG. 1. Note that while FIG. 11 illustratesvarious components of a computer system, it is not intended to representany particular architecture or manner of interconnecting the componentsas such details are not germane to the present invention. It will alsobe appreciated that network computers and other data processing systemsor other consumer electronic devices, which have fewer components orperhaps more components, may also be used with the present invention.

As shown in FIG. 11, the computer system 1100, which is a form of a dataprocessing system, includes a bus 1103 which is coupled to amicroprocessor(s) 1105 and a ROM (Read Only Memory) 1107 and volatileRAM 1109 and a non-volatile memory 1111. The microprocessor 1105 mayretrieve the instructions from the memories 1107, 1109, 1111 and executethe instructions to perform operations described above. The bus 1103interconnects these various components together and also interconnectsthese components 1105, 1107, 1109, and 1111 to a display controller anddisplay device 1117 and to peripheral devices such as input/output (I/O)devices which may be mice, keyboards, modems, network interfaces,printers and other devices which are well known in the art. In oneembodiment, the system 1100 includes a plurality of network interfacesof the same or different type (e.g., Ethernet copper interface, Ethernetfiber interfaces, wireless, and/or other types of network interfaces).In this embodiment, the system 1100 can include a forwarding engine toforward network date received on one interface out another interface.

Typically, the input/output devices 1115 are coupled to the systemthrough input/output controllers 1113. The volatile RAM (Random AccessMemory) 1109 is typically implemented as dynamic RAM (DRAM), whichrequires power continually in order to refresh or maintain the data inthe memory.

The mass storage 1111 is typically a magnetic hard drive or a magneticoptical drive or an optical drive or a DVD ROM/RAM or a flash memory orother types of memory systems, which maintains data (e.g. large amountsof data) even after power is removed from the system. Typically, themass storage 1111 will also be a random access memory although this isnot required. While FIG. 11 shows that the mass storage 1111 is a localdevice coupled directly to the rest of the components in the dataprocessing system, it will be appreciated that the present invention mayutilize a non-volatile memory which is remote from the system, such as anetwork storage device which is coupled to the data processing systemthrough a network interface such as a modem, an Ethernet interface or awireless network. The bus 1103 may include one or more buses connectedto each other through various bridges, controllers and/or adapters as iswell known in the art.

Portions of what was described above may be implemented with logiccircuitry such as a dedicated logic circuit or with a microcontroller orother form of processing core that executes program code instructions.Thus processes taught by the discussion above may be performed withprogram code such as machine-executable instructions that cause amachine that executes these instructions to perform certain functions.In this context, a “machine” may be a machine that converts intermediateform (or “abstract”) instructions into processor specific instructions(e.g., an abstract execution environment such as a “process virtualmachine” (e.g., a Java Virtual Machine), an interpreter, a CommonLanguage Runtime, a high-level language virtual machine, etc.), and/or,electronic circuitry disposed on a semiconductor chip (e.g., “logiccircuitry” implemented with transistors) designed to executeinstructions such as a general-purpose processor and/or aspecial-purpose processor. Processes taught by the discussion above mayalso be performed by (in the alternative to a machine or in combinationwith a machine) electronic circuitry designed to perform the processes(or a portion thereof) without the execution of program code.

The present invention also relates to an apparatus for performing theoperations described herein. This apparatus may be specially constructedfor the required purpose, or it may comprise a general-purpose computerselectively activated or reconfigured by a computer program stored inthe computer. Such a computer program may be stored in a computerreadable storage medium, such as, but is not limited to, any type ofdisk including floppy disks, optical disks, CD-ROMs, andmagnetic-optical disks, read-only memories (ROMs), RAMs, EPROMs,EEPROMs, magnetic or optical cards, or any type of media suitable forstoring electronic instructions, and each coupled to a computer systembus.

A machine readable medium includes any mechanism for storing ortransmitting information in a form readable by a machine (e.g., acomputer). For example, a machine readable medium includes read onlymemory (“ROM”); random access memory (“RAM”); magnetic disk storagemedia; optical storage media; flash memory devices; etc.

An article of manufacture may be used to store program code. An articleof manufacture that stores program code may be embodied as, but is notlimited to, one or more memories (e.g., one or more flash memories,random access memories (static, dynamic or other)), optical disks,CD-ROMs, DVD ROMs, EPROMs, EEPROMs, magnetic or optical cards or othertype of machine-readable media suitable for storing electronicinstructions. Program code may also be downloaded from a remote computer(e.g., a server) to a requesting computer (e.g., a client) by way ofdata signals embodied in a propagation medium (e.g., via a communicationlink (e.g., a network connection)).

FIG. 12 is a block diagram of one embodiment of an exemplary networkelement 1200 that synchronizes an address resolution table of a networkelement table of a network element. In FIG. 12, the backplane 1206couples to the line cards 1202A-N and controller cards 1204A-B. While inone embodiment, the controller cards 1204A-B control the processing ofthe traffic by the line cards 1202A-N, in alternate embodiments, thecontroller cards 1204A-B, perform the same and/or different functions(e.g., synchronizing an address resolution table of a network elementtable of a network element, etc.). In this embodiment, one, some, or allof the controller cards 1204A-B includes the synchronize module tosynchronizes an address resolution table or a ARP sync and recoverymodule to synchronize an address resolution table between multiple CPUsin this network element 1200, such as the synchronize module 206A-B asdescribed in FIG. 2 above or ARP sync and recovery module 506A-B asdescribed in FIG. 5 above. In one embodiment, the line cards 1202A-Nprocess and forward traffic according to the network policies receivedfrom controller cards the 1204A-B. It should be understood that thearchitecture of the network element 1200 illustrated in FIG. 12 isexemplary, and different combinations of cards may be used in otherembodiments of the invention.

The preceding detailed descriptions are presented in terms of algorithmsand symbolic representations of operations on data bits within acomputer memory. These algorithmic descriptions and representations arethe tools used by those skilled in the data processing arts to mosteffectively convey the substance of their work to others skilled in theart. An algorithm is here, and generally, conceived to be aself-consistent sequence of operations leading to a desired result. Theoperations are those requiring physical manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of electrical or magnetic signals capable of being stored,transferred, combined, compared, and otherwise manipulated. It hasproven convenient at times, principally for reasons of common usage, torefer to these signals as bits, values, elements, symbols, characters,terms, numbers, or the like.

It should be kept in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the above discussion, itis appreciated that throughout the description, discussions utilizingterms such as “receiving,” “generating,” “determining,” “performing,”“forwarding,” “storing,” “identifying,” “updating,” “synchronizing,”“reconciling,” “starting,” or the like, refer to the action andprocesses of a computer system, or similar electronic computing device,that manipulates and transforms data represented as physical(electronic) quantities within the computer system's registers andmemories into other data similarly represented as physical quantitieswithin the computer system memories or registers or other suchinformation storage, transmission or display devices.

The processes and displays presented herein are not inherently relatedto any particular computer or other apparatus. Various general-purposesystems may be used with programs in accordance with the teachingsherein, or it may prove convenient to construct a more specializedapparatus to perform the operations described. The required structurefor a variety of these systems will be evident from the descriptionbelow. In addition, the present invention is not described withreference to any particular programming language. It will be appreciatedthat a variety of programming languages may be used to implement theteachings of the invention as described herein.

The foregoing discussion merely describes some exemplary embodiments ofthe present invention. One skilled in the art will readily recognizefrom such discussion, the accompanying drawings and the claims thatvarious modifications can be made without departing from the spirit andscope of the invention.

1.-8. (canceled)
 9. A non-transitory machine-readable medium havingexecutable instructions to cause one or more processing units perform amethod to boot-up of a first processing unit in a network element, themethod comprising: periodically synchronizing software addressresolution information between the first and a second processing unit ofthe network element; receiving an indication of the boot-up of the firstprocessing unit; determining differences between a hardware addressresolution information and the software address resolution information;reconciling the differences so that the hardware and the softwareaddress information includes the same address resolution entries; andstarting the address resolution service on the first processing unit.10. The non-transitory machine-readable medium of claim 9, wherein theaddress resolution information is Address Resolution Protocolinformation.
 11. The non-transitory machine-readable medium of claim 9,wherein the synchronizing software address resolution informationcomprises: receiving the address resolution information; and installingthe address resolution information.
 12. The non-transitorymachine-readable medium of claim 9, wherein reconciling the differencescomprises: retrieving the hardware and software address information; andfixing differences between the hardware and software addressinformation. 13-20. (canceled)
 21. The non-transitory machine-readablemedium of claim 9, wherein the network element is implemented as a firsthop gateway.
 22. The non-transitory machine-readable medium of claim 9,wherein the network element provides access to a network layer for thefirst and second processing units from a different network layer. 23.The non-transitory machine-readable medium of claim 22, wherein thenetwork element provides access to a layer 3 network layer for the firstand second processing units from a layer 2 network layer.
 24. Thenon-transitory machine-readable medium of claim 9, wherein theresolution table is a serialized table.
 25. The non-transitorymachine-readable medium of claim 9, wherein the network element is anetwork element peer of another network element in a multi-chassis linkaggregation topology.
 26. A network element comprising: first and secondprocessing units; one or more processors programmed and configured toperform: periodically synchronizing software address resolutioninformation between the first and a second processing unit of thenetwork element; receiving an indication of the boot-up of the firstprocessing unit; determining differences between a hardware addressresolution information and the software address resolution information;reconciling the differences so that the hardware and the softwareaddress information includes the same address resolution entries; andstarting the address resolution service on the first processing unit.27. The network element of claim 26 wherein the address resolutioninformation is Address Resolution Protocol information.
 28. The networkelement of claim 26, wherein the synchronizing software addressresolution information comprises: receiving the address resolutioninformation; and installing the address resolution information.
 29. Thenetwork element of claim 26, wherein reconciling the differencescomprises: retrieving the hardware and software address information; andfixing differences between the hardware and software addressinformation.
 30. The network element of claim 26, wherein the networkelement is implemented as a first hop gateway.
 31. The network elementof claim 26, wherein the network element provides access to a networklayer for the first and second processing units from a different networklayer.
 32. The network element of claim 26, wherein the network elementprovides access to a layer 3 network layer for the first and secondprocessing units from a layer 2 network layer.
 33. The network elementof claim 26, wherein the resolution table is a serialized table.
 34. Thenetwork element of claim 26, wherein the network element is a networkelement peer of another network element in a multi-chassis linkaggregation topology.
 35. A method of starting an address resolutionservice, the method comprising: periodically synchronizing softwareaddress resolution information between a first and a second processingunit of a network element; receiving an indication of the boot-up of thefirst processing unit; determining differences between a hardwareaddress resolution information and the software address resolutioninformation; reconciling the differences so that the hardware and thesoftware address information includes the same address resolutionentries; and starting the address resolution service on the firstprocessing unit.
 36. The method of claim 35, wherein the addressresolution information is Address Resolution Protocol information. 37.The method of claim 35, wherein the synchronizing software addressresolution information comprises: receiving the address resolutioninformation; and installing the address resolution information.
 38. Themethod of claim 35, wherein reconciling the differences comprises:retrieving the hardware and software address information; and fixingdifferences between the hardware and software address information. 39.The method of claim 35, wherein the network element is implemented as afirst hop gateway.
 40. The method of claim 35, wherein the networkelement provides access to a network layer for the first and secondprocessing units from a different network layer.
 41. The method of claim40, wherein the network element provides access to a layer 3 networklayer for the first and second processing units from a layer 2 networklayer.
 42. The method of claim 35, wherein the resolution table is aserialized table.
 43. The method of claim 35, wherein the networkelement is a network element peer of another network element in amulti-chassis link aggregation topology.
 44. A method of starting anaddress resolution service, the method comprising: periodicallysynchronizing software address resolution information between an firstprocessing unit and a second processing unit of a network element, thefirst processing unit actively processing an address resolution service;receiving an indication of the failure of the active processing unit;determining differences between a hardware address resolutioninformation and the software address resolution information; reconcilingthe differences so that the hardware and the software addressinformation includes the same address resolution entries; and resumingthe address resolution service on the second processing unit.
 45. Themethod of claim 44, wherein the network element is a network elementpeer of another network element in a multi-chassis link aggregationtopology.